Real-world feedback network for feedback recipients

ABSTRACT

An online feedback network provides feedback from contributors to a feedback recipient for a project. A request modifier may receive a default request from a data source and allow the feedback recipients to use the default request, modify the default request and/or allow the feedback recipient to create an initial request in requesting feedback for each feedback recipient&#39;s project from the contributors. The request modifier may also modify the default or initial request so that the request from the feedback recipient receives a desired volume, type, source or network of feedback. For instance the request modifier may increase the number of contributors receiving the request or simplify the type of requested feedback in order to increase the volume of feedback received by the feedback recipients based on previous requests for feedback and the volume of feedback received by the past requests. The submitted request may be stored for future use.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority from U.S. patentapplication Ser. No. 15/714,645, filed under the same title on Sep. 25,2017, the entire contents of which are incorporated herein by reference.

FIELD OF THE INVENTION

This disclosure relates to the field of systems and methods that allowan instructor and/or a learner to shape and request online feedbackreceived over a computer network from a plurality of contributors.

SUMMARY OF THE INVENTION

The present invention provides systems and methods comprising one ormore server hardware computing devices or client hardware computingdevices, communicatively coupled to a network, and each comprising atleast one processor executing specific computer-executable instructionswithin a memory that, when executed, cause the system to allow aninstructor and/or a learner to shape and request feedback for a projectfrom a plurality of contributors.

The invention is a method for an online educational institute to providefeedback for a project from a plurality of contributors that are part ofa feedback network to a learner and an instructor. The onlineeducational institute may comprise computer code running on one or morehardware servers and may access to one or more hardware databases. Thecomputer code may be, at least, a request modifier and a feedbackaggregator.

The request modifier may electronically transmit a first interface to aclient device of an instructor configured to allow the client device ofthe instructor to enter a default request for feedback for the projectfrom client devices of the plurality of contributors to the learner. Thedefault request may include, as non-limiting examples, a feedback duedate or a project due date, a project title or a project description, asuggested feedback type, a desired volume of feedback and a plurality ofgroups, each group comprising a plurality of potential contributors.

The request modifier may assign a character string to the defaultrequest entered by the instructor. The instructor may communicate,preferably via the client devices of the instructor and the learners,the character string assigned to the default request for feedback fromthe contributors.

The request modifier may electronically receive the character stringfrom the client device of the learner. The request modifier mayelectronically transmit a second interface to the client device of thelearner. In some embodiments, the learner is not given an option tochange the default request for feedback as entered by the instructor.The default request is submitted to the contributors.

In other embodiments, the interface may be configured to allow theclient device of the learner to enter an initial request (possiblealterations or modifications from the default request) for the feedbackfor the project from client devices of the plurality of contributors tothe learner. As non-limiting examples, the initial request may comprisethe feedback due date or the project due date, the project title or theproject description, the suggested feedback type and/or the desiredvolume of feedback and the plurality of groups with each groupcomprising a plurality of potential contributors.

The request modifier may electronically receive a plurality of pastrequests and a corresponding plurality of past feedback from a hardwaredatabase. The request modifier may use the past requests andcorresponding past feedback to determine a plurality of most similarpast requests to the initial request in the plurality of past requests.The request modifier may use the past volume of feedback from the mostsimilar past requests to determine an average volume of feedback for thesimilar past requests.

The request modifier may assign a predicted volume of feedback for thedefault request or initial request based on an average volume of pastfeedback for the plurality of most similar past requests.

The request modifier may modify the default request or the initialrequest to generate a modified request that is preferably predicted toreceive a volume of feedback closer to the desired volume of feedbackthan the predicted volume of feedback for the default request or theinitial request. The request modifier may make adjustments, such aschanging the type of desired feedback or the number of contributorsreceiving the request for feedback, to create a modified request thatwill receive a desired volume of feedback as selected by the instructoror learner.

The request modifier may electronically transmit the modified request tothe plurality of contributors for feedback. One or more of thecontributors receiving the submitted request for feedback may accept therequest and/or provide feedback on the project using an interfacepresented to the contributors.

The feedback aggregator may electronically receive the plurality offeedback from the one or more responding contributors. The feedbackaggregator may electronically transmit an interface displaying theplurality of feedback for the project from the plurality of contributorsto the client device of the learner and a client device of aninstructor.

The above features and advantages of the present invention will bebetter understood from the following detailed description taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system level block diagram for a computerenvironment that allows an instructor or a user on a client device toshape and request feedback over a computer network from a plurality ofcontributors on client devices.

FIG. 2 illustrates a system level block diagram for a computer systemthat may be used in part to allow an instructor and/or a user on aclient device to shape and request feedback over a computer network froma plurality of contributors on client devices.

FIG. 3 illustrates a block diagram of a system that allows an instructorand/or a user on a client device to shape and request feedback over acomputer network from a plurality of contributors on client devices. Arequest modifier may also modify a request for feedback from theinstructor or the learner so as to provide a desired volume of feedback.

FIG. 4 illustrates a flow chart of a method that may be used to allowsan instructor and/or a user on a client device to shape and requestfeedback over a computer network from a plurality of contributors onclient devices. A request modifier may also modify a request forfeedback from the instructor or the learner so as to provide a desiredvolume of feedback.

FIG. 5 illustrates a block diagram of a system that allows an instructorand/or a user on a client device to shape and request feedback over acomputer network from a plurality of contributors on client devices.

FIG. 6 illustrates a flow chart of a method that may be used to allowsan instructor and/or a user on a client device to shape and requestfeedback over a computer network from a plurality of contributors onclient devices.

FIG. 7 illustrates a possible interface that may be displayed by arequest modifier of an online educational institute to a client deviceof an instructor to enter a default request or a client device of alearner to enter an initial request.

FIG. 8 illustrates a possible interface for a learner to enter acharacter string that will auto populate the default request entered bythe instructor into the interface displayed on the client device of thelearner.

FIG. 9 illustrates a possible interface that may be electronicallycommunicated to a client device of a contributor requesting that thecontributor provide feedback to the described project of the identifiedlearner and allowing the contributor to accept or reject the request.

FIG. 10 illustrates a possible interface displayed on a client devicefor a contributor to enter feedback for the project of the learner.

FIG. 11 illustrates a possible interface for a feedback aggregator tosummarize the feedback received from a plurality of contributors to aclient device of an instructor and a learner.

FIG. 12 illustrates a possible interface that may be displayed by theonline educational institute on a client device of a learner to receivefeedback or a client device of a contributor to enter feedback on aproject.

FIG. 13 illustrates a possible interface for the instructor or learnerto receive on a client device an aggregate view and/or summary view ofthe feedback received from a plurality of contributors.

FIG. 14 illustrates a possible interface for the instructor or learnerto receive on a client device that compares the learner's skills in aplurality of areas with the known necessary skills for different careeror employment positions.

DETAILED DESCRIPTION

The present inventions will now be discussed in detail with regard tothe attached drawing figures that were briefly described above. In thefollowing description, numerous specific details are set forthillustrating the Applicant's best mode for practicing the invention andenabling one of ordinary skill in the art to make and use the invention.It will be obvious, however, to one skilled in the art that the presentinvention may be practiced without many of these specific details. Inother instances, well-known machines, structures, and method steps havenot been described in particular detail in order to avoid unnecessarilyobscuring the present invention. Unless otherwise indicated, like partsand method steps are referred to with like reference numerals.

Network

FIG. 1 illustrates a non-limiting example distributed computingenvironment 100, which includes one or more computer server computingdevices 102, one or more client computing devices 106, and othercomponents that may implement certain embodiments and features describedherein. Other devices, such as specialized sensor devices, etc., mayinteract with client 106 and/or server 102. The server 102, client 106,or any other devices may be configured to implement a client-servermodel or any other distributed computing architecture.

Server 102, client 106, and any other disclosed devices may becommunicatively coupled via one or more communication networks 120.Communication network 120 may be any type of network known in the artsupporting data communications. As non-limiting examples, network 120may be a local area network (LAN; e.g., Ethernet, Token-Ring, etc.), awide-area network (e.g., the Internet), an infrared or wireless network,a public switched telephone networks (PSTNs), a virtual network, etc.Network 120 may use any available protocols, such as (e.g., transmissioncontrol protocol/Internet protocol (TCP/IP), systems networkarchitecture (SNA), Internet packet exchange (IPX), Secure Sockets Layer(SSL), Transport Layer Security (TLS), Hypertext Transfer Protocol(HTTP), Secure Hypertext Transfer Protocol (HTTPS), Institute ofElectrical and Electronics (IEEE) 802.11 protocol suite or otherwireless protocols, and the like.

Servers/Clients

The embodiments shown in FIGS. 1-2 are thus one example of a distributedcomputing system and is not intended to be limiting. The subsystems andcomponents within the server 102 and client devices 106 may beimplemented in hardware, firmware, software, or combinations thereof.Various different subsystems and/or components 104 may be implemented onserver 102. Users operating the client devices 106 may initiate one ormore client applications to use services provided by these subsystemsand components. Various different system configurations are possible indifferent distributed computing systems 100 and content distributionnetworks. Server 102 may be configured to run one or more serversoftware applications or services, for example, web-based or cloud-basedservices, to support content distribution and interaction with clientdevices 106. Users operating client devices 106 may in turn utilize oneor more client applications (e.g., virtual client applications) tointeract with server 102 to utilize the services provided by thesecomponents. Client devices 106 may be configured to receive and executeclient applications over one or more networks 120. Such clientapplications may be web browser based applications and/or standalonesoftware applications, such as mobile device applications. Clientdevices 106 may receive client applications from server 102 or fromother application providers (e.g., public or private applicationstores).

Security

As shown in FIG. 1, various security and integration components 108 maybe used to manage communications over network 120 (e.g., a file-basedintegration scheme or a service-based integration scheme). Security andintegration components 108 may implement various security features fordata transmission and storage, such as authenticating users orrestricting access to unknown or unauthorized users.

As non-limiting examples, these security components 108 may comprisededicated hardware, specialized networking components, and/or software(e.g., web servers, authentication servers, firewalls, routers,gateways, load balancers, etc.) within one or more data centers in oneor more physical location and/or operated by one or more entities,and/or may be operated within a cloud infrastructure.

In various implementations, security and integration components 108 maytransmit data between the various devices in the content distributionnetwork 100. Security and integration components 108 also may use securedata transmission protocols and/or encryption (e.g., File TransferProtocol (FTP), Secure File Transfer Protocol (SFTP), and/or Pretty GoodPrivacy (PGP) encryption) for data transfers, etc.).

In some embodiments, the security and integration components 108 mayimplement one or more web services (e.g., cross-domain and/orcross-platform web services) within the content distribution network100, and may be developed for enterprise use in accordance with variousweb service standards (e.g., the Web Service Interoperability (WS-I)guidelines). For example, some web services may provide secureconnections, authentication, and/or confidentiality throughout thenetwork using technologies such as SSL, TLS, HTTP, HTTPS, WS-Securitystandard (providing secure SOAP messages using XML encryption), etc. Inother examples, the security and integration components 108 may includespecialized hardware, network appliances, and the like (e.g.,hardware-accelerated SSL and HTTPS), possibly installed and configuredbetween servers 102 and other network components, for providing secureweb services, thereby allowing any external devices to communicatedirectly with the specialized hardware, network appliances, etc.

Data Stores (Databases)

Computing environment 100 also may include one or more data stores 110,possibly including and/or residing on one or more back-end servers 112,operating in one or more data centers in one or more physical locations,and communicating with one or more other devices within one or morenetworks 120. In some cases, one or more data stores 110 may reside on anon-transitory storage medium within the server 102. In certainembodiments, data stores 110 and back-end servers 112 may reside in astorage-area network (SAN). Access to the data stores may be limited ordenied based on the processes, user credentials, and/or devicesattempting to interact with the data store.

Computer System

With reference now to FIG. 2, a block diagram of an illustrativecomputer system is shown. The system 200 may correspond to any of thecomputing devices, clients, hardware servers or servers of the network100, or any other computing devices described herein. In this example,computer system 200 includes processing units 204 that communicate witha number of peripheral subsystems via a bus subsystem 202. Theseperipheral subsystems include, for example, a storage subsystem 210, anI/O subsystem 226, and a communications subsystem 232.

Processors

One or more processing units 204 may be implemented as one or moreintegrated circuits (e.g., a conventional micro-processor ormicrocontroller), and controls the operation of computer system 200.These processors may include single core and/or multicore (e.g., quadcore, hexa-core, octo-core, ten-core, etc.) processors and processorcaches. These processors 204 may execute a variety of resident softwareprocesses embodied in program code, and may maintain multipleconcurrently executing programs or processes. Processor(s) 204 may alsoinclude one or more specialized processors, (e.g., digital signalprocessors (DSPs), outboard, graphics application-specific, and/or otherprocessors).

Buses

Bus subsystem 202 provides a mechanism for intended communicationbetween the various components and subsystems of computer system 200.Although bus subsystem 202 is shown schematically as a single bus,alternative embodiments of the bus subsystem may utilize multiple buses.Bus subsystem 202 may include a memory bus, memory controller,peripheral bus, and/or local bus using any of a variety of busarchitectures (e.g. Industry Standard Architecture (ISA), Micro ChannelArchitecture (MCA), Enhanced ISA (EISA), Video Electronics StandardsAssociation (VESA), and/or Peripheral Component Interconnect (PCI) bus,possibly implemented as a Mezzanine bus manufactured to the IEEE P1386.1standard).

Input/Output

I/O subsystem 226 may include device controllers 228 for one or moreuser interface input devices and/or user interface output devices,possibly integrated with the computer system 200 (e.g., integratedaudio/video systems, and/or touchscreen displays), or may be separateperipheral devices which are attachable/detachable from the computersystem 200. Input may include keyboard or mouse input, audio input(e.g., spoken commands), motion sensing, gesture recognition (e.g., eyegestures), etc.

Input

As non-limiting examples, input devices may include a keyboard, pointingdevices (e.g., mouse, trackball, and associated input), touchpads, touchscreens, scroll wheels, click wheels, dials, buttons, switches, keypad,audio input devices, voice command recognition systems, microphones,three dimensional (3D) mice, joysticks, pointing sticks, gamepads,graphic tablets, speakers, digital cameras, digital camcorders, portablemedia players, webcams, image scanners, fingerprint scanners, barcodereaders, 3D scanners, 3D printers, laser rangefinders, eye gaze trackingdevices, medical imaging input devices, MIDI keyboards, digital musicalinstruments, and the like.

Output

In general, use of the term “output device” is intended to include allpossible types of devices and mechanisms for outputting information fromcomputer system 200 to a user or other computer. For example, outputdevices may include one or more display subsystems and/or displaydevices that visually convey text, graphics and audio/video information(e.g., cathode ray tube (CRT) displays, flat-panel devices, liquidcrystal display (LCD) or plasma display devices, projection devices,touch screens, etc.), and/or non-visual displays such as audio outputdevices, etc. As non-limiting examples, output devices may include,indicator lights, monitors, printers, speakers, headphones, automotivenavigation systems, plotters, voice output devices, modems, etc.

Memory or Storage Media

Computer system 200 may comprise one or more storage subsystems 210,comprising hardware and software components used for storing data andprogram instructions, such as system memory 218 and computer-readablestorage media 216.

System memory 218 and/or computer-readable storage media 216 may storeprogram instructions that are loadable and executable on processor(s)204. For example, system memory 218 may load and execute an operatingsystem 224, program data 222, server applications, client applications220, Internet browsers, mid-tier applications, etc.

System memory 218 may further store data generated during execution ofthese instructions. System memory 218 may be stored in volatile memory(e.g., random access memory (RAM) 212, including static random accessmemory (SRAM) or dynamic random access memory (DRAM)). RAM 212 maycontain data and/or program modules that are immediately accessible toand/or operated and executed by processing units 204.

System memory 218 may also be stored in non-volatile storage drives 214(e.g., read-only memory (ROM), flash memory, etc.) For example, a basicinput/output system (BIOS), containing the basic routines that help totransfer information between elements within computer system 200 (e.g.,during start-up) may typically be stored in the non-volatile storagedrives 214.

Computer Readable Storage Media

Storage subsystem 210 also may include one or more tangiblecomputer-readable storage media 216 for storing the basic programmingand data constructs that provide the functionality of some embodiments.For example, storage subsystem 210 may include software, programs, codemodules, instructions, etc., that may be executed by a processor 204, inorder to provide the functionality described herein. Data generated fromthe executed software, programs, code, modules, or instructions may bestored within a data storage repository within storage subsystem 210.

Storage subsystem 210 may also include a computer-readable storage mediareader connected to computer-readable storage media 216.Computer-readable storage media 216 may contain program code, orportions of program code. Together and, optionally, in combination withsystem memory 218, computer-readable storage media 216 maycomprehensively represent remote, local, fixed, and/or removable storagedevices plus storage media for temporarily and/or more permanentlycontaining, storing, transmitting, and retrieving computer-readableinformation.

Computer-readable storage media 216 may include any appropriate mediaknown or used in the art, including storage media and communicationmedia, such as but not limited to, volatile and non-volatile, removableand non-removable media implemented in any method or technology forstorage and/or transmission of information. This can include tangiblecomputer-readable storage media such as RAM, ROM, electronicallyerasable programmable ROM (EEPROM), flash memory or other memorytechnology, CD-ROM, digital versatile disk (DVD), or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or other tangible computer readablemedia. This can also include nontangible computer-readable media, suchas data signals, data transmissions, or any other medium which can beused to transmit the desired information and which can be accessed bycomputer system 200.

By way of example, computer-readable storage media 216 may include ahard disk drive that reads from or writes to non-removable, nonvolatilemagnetic media, a magnetic disk drive that reads from or writes to aremovable, nonvolatile magnetic disk, and an optical disk drive thatreads from or writes to a removable, nonvolatile optical disk such as aCD ROM, DVD, and Blu-Ray® disk, or other optical media.Computer-readable storage media 216 may include, but is not limited to,Zip® drives, flash memory cards, universal serial bus (USB) flashdrives, secure digital (SD) cards, DVD disks, digital video tape, andthe like. Computer-readable storage media 216 may also include,solid-state drives (SSD) based on non-volatile memory such asflash-memory based SSDs, enterprise flash drives, solid state ROM, andthe like, SSDs based on volatile memory such as solid state RAM, dynamicRAM, static RAM, DRAM-based SSDs, magneto-resistive RAM (MRAM) SSDs, andhybrid SSDs that use a combination of DRAM and flash memory based SSDs.The disk drives and their associated computer-readable media may providenon-volatile storage of computer-readable instructions, data structures,program modules, and other data for computer system 200.

Communication Interface

Communications subsystem 232 may provide a communication interface fromcomputer system 200 and external computing devices via one or morecommunication networks, including local area networks (LANs), wide areanetworks (WANs) (e.g., the Internet), and various wirelesstelecommunications networks. As illustrated in FIG. 2, thecommunications subsystem 232 may include, for example, one or morenetwork interface controllers (NICs) 234, such as Ethernet cards,Asynchronous Transfer Mode NICs, Token Ring NICs, and the like, as wellas one or more wireless communications interfaces 236, such as wirelessnetwork interface controllers (WNICs), wireless network adapters, andthe like. Additionally and/or alternatively, the communicationssubsystem 232 may include one or more modems (telephone, satellite,cable, ISDN), synchronous or asynchronous digital subscriber line (DSL)units, Fire Wire® interfaces, USB® interfaces, and the like.Communications subsystem 236 also may include radio frequency (RF)transceiver components for accessing wireless voice and/or data networks(e.g., using cellular telephone technology, advanced data networktechnology, such as 3G, 4G or EDGE (enhanced data rates for globalevolution), WiFi (IEEE 802.11 family standards, or other mobilecommunication technologies, or any combination thereof), globalpositioning system (GPS) receiver components, and/or other components.

Input Output Streams Etc.

In some embodiments, communications subsystem 232 may also receive inputcommunication in the form of structured and/or unstructured data feeds,event streams, event updates, and the like, on behalf of one or moreusers who may use or access computer system 200. For example,communications subsystem 232 may be configured to receive data feeds inreal-time from users of social networks and/or other communicationservices, web feeds such as Rich Site Summary (RSS) feeds, and/orreal-time updates from one or more third party information sources(e.g., data aggregators). Additionally, communications subsystem 232 maybe configured to receive data in the form of continuous data streams,which may include event streams of real-time events and/or event updates(e.g., sensor data applications, financial tickers, network performancemeasuring tools, clickstream analysis tools, automobile trafficmonitoring, etc.). Communications subsystem 232 may output suchstructured and/or unstructured data feeds, event streams, event updates,and the like to one or more data stores that may be in communicationwith one or more streaming data source computers coupled to computersystem 200.

Connect Components to System

The various physical components of the communications subsystem 232 maybe detachable components coupled to the computer system 200 via acomputer network, a FireWire® bus, or the like, and/or may be physicallyintegrated onto a motherboard of the computer system 200. Communicationssubsystem 232 also may be implemented in whole or in part by software.

Other Variations

Due to the ever-changing nature of computers and networks, thedescription of computer system 200 depicted in the figure is intendedonly as a specific example. Many other configurations having more orfewer components than the system depicted in the figure are possible.For example, customized hardware might also be used and/or particularelements might be implemented in hardware, firmware, software, or acombination. Further, connection to other computing devices, such asnetwork input/output devices, may be employed. Based on the disclosureand teachings provided herein, a person of ordinary skill in the artwill appreciate other ways and/or methods to implement the variousembodiments.

Referring to FIGS. 3-6, systems and methods are illustrated for aninstructor 370, i.e., a data source 370 and a learner 350, i.e., afeedback recipient 350, in an online educational institute 300, i.e., anonline digital feedback network 300, to control the sources, type andquantity of feedback that the instructor 370 and learner 350 receivefrom one or more contributors 360 regarding a project the learner 350 isperforming as a class assignment. It should be appreciated that while,in some embodiments, the instructor 370 and learners 350 may physicallymeet in a brick & mortar building or alternatively only in a virtualonline class, the requests for feedback to the contributors 360 and thefeedback from the contributors 360 to the learners 350 must be receivedvia an online digital electronic computer network (such as theInternet).

The claimed invention is advantageous for the learner 350 as the learner350 is able to manage the sources, type (quality) and quantity offeedback the learner 350 receives from one or more contributors 360. Theimproved feedback enhances the learning experience of the learner andmay also provide the learner (preferably with the consent of thecontributors 360) with a portfolio of projects that have been reviewedby a number of professionals in the same field as the project. Theonline educational institute 300, instructor 370 and learner 350 may alluse the project and the feedback as marketing and promotional tools. Inpreferred embodiments, authorizations and/or notifications have beenobtained from the instructor 370, the learner 350 and/or thecontributors 360 whose material is being used as marketing orpromotional tools.

The claimed invention is also advantageous for the instructor 370 as thefeedback from the contributors 360 enhances the learning experience ofthe learners 350 in the instructor's class, i.e., learning group orgroup. The instructor 370 may request that projects only be turned inwith specified types or volumes of feedback, may request that a draft,feedback on the draft and the final updated project be turned in orframe the requested feedback from the contributors 360. In addition, theaggregate feedback from the contributors 360 may be used by theinstructor 370 in evaluating the course, using the feedback as insightsfrom other experts in the field. Thus, the instructor may update thecourse based on the feedback from the contributors 360.

The claimed invention is also advantageous for the contributors 360 asthe feedback provided by the contributors 360 enhances the contributors'digital reputation and builds networking contacts with the learners 350.The contributors 360 also see the latest academic work being done intheir field and how the latest batch of learners 350 (students) areaddressing the current issues.

An online educational institute 300 may be a legal entity in thebusiness of providing educational classes, such as traditional and/oronline digital classes, that include assigning projects. The onlineeducational institute 300 comprises computer code running on one or morehardware servers 112 and other computer hardware, such as routers, thatpermit the online educational institute 300 to electronicallycommunicate with their associated learners 350, contributors 360 andinstructors 370. Each learner 350, contributor and instructor 370 mayuse a client 108 or client device 108 (such as a smartphone, tablet,laptop or desktop computer) to electronically communicate with theonline educational institute 300.

The computer code of the online educational institute 300 may provideany number of different desired services. In one embodiment, thecomputer code of the online educational institute 300 comprises arequest modifier 320 and a feedback aggregator 330. The request modifier320 and feedback aggregator 330 may comprise one or more computerprograms and/or computer software modules running on the one or morehardware servers 112 of the online educational institute 300.

The request modifier 320 may receive from a learner 350 an initialrequest for feedback from a plurality of contributors 360 and,optionally, a desired volume of feedback from the contributors 360.(Step 400) In other embodiments, a maximum number of contributors 360providing feedback may be included and/or a date after which feedbackcan no longer be received from the contributors 360 may also be includedwith the request to the request modifier 320. The request modifier 320may modify the initial request created by the learner 350 for feedbackfrom a plurality of contributors 360. (Step 410) The request modifier320 may make the modifications for any desired reason, such as therequest from the learner 350 is accompanied by a desired volume (numberof contributors 360 and/or amount of feedback) of feedback from thecontributors 360.

The feedback aggregator 330 may be used to receive, aggregate andtransmit the feedback from the contributors 360 to the learner 350and/or instructor 370. The feedback aggregator 330 may store the requestsubmitted to the contributors 360 and the feedback (or summary of thefeedback, such as the number of contributors 360 that responded and/orthe quantity of the feedback) in a hardware database 110 (also referredto as a data store 110). The request modifier 320 for future requestsfrom the same or different learners 350 may use the request submitted tothe contributors 360 and the feedback stored in the hardware database110 to allow the request modifier 320 to modify requests so that themodified request submitted to the contributors 360 receives a desiredvolume of feedback as indicated by the learner 350. In some embodiments,the request modifier 320 may provide suggestions and/or defaults in auser interface (UI) as part of a user experience (UX) to scaffold therequesting process such that learners 350 are more likely to choosesettings aligned with their feedback needs, i.e., requests withattributes predicted to yield the desired feedback, either by type,volume and/or deadline.

The online educational institute 300 may also comprise a hardwaredatabase 110 that allows the storage and retrieval of past requests (andthe date created or submitted to the contributors 360) and the pastrequests' corresponding feedback (responses). This data may be enteredinto the hardware database 110 by the feedback aggregator 330 afterevery learner request receives feedback. In addition, the requestmodifier 320 may read and use the request and corresponding feedback tomodify requests, either the request itself and/or the request makingexperience, from learners 350 for feedback from contributors 360 on theassumption that similar requests will receive a similar volume offeedback from contributors 360.

In another embodiment, the request modifier 320 of the online educationinstitute 300 may transmit a first interface to a client device 108 ofan instructor 370. The first interface may be configured to allow theclient device 108 of the instructor 370 to enter a default request forthe feedback for a project from client devices 108 of one or morecontributors 360. The default request from the instructor 370 may beintended by the instructor 370 to be used by all of the learners in aclass working on the project. The default request may comprise anyfactors desired by the instructor 370 for the project. As non-limitingexamples, the default request (and all of the other described requestsbelow) may comprise a feedback due date or a project due date, a projecttitle or a project description, a suggested/required feedback type, adesired volume of feedback and/or one or more groups, with each groupcomprising a plurality of potential contributors 360.

The suggested feedback type may comprise any category of feedbackdesired by the instructor 370 and/or learner. As non-limiting examples,the suggested feedback type may be a scoring method of a positive or anegative reaction (thumbs up or thumbs down), a rating on a scale,ratings to various factors listed in a rubric, a drafting comments in asuggestions area and/or an annotated response to the material of theproject. In addition, the suggested feedback type may ask for feedback(using any of the above mentioned scoring methods) on various aspects orfactors of the project. As non-limiting examples, the aspects or factorsof the project may comprise a writing skill, creativity, informationliteracy, visual design and/or critical thinking demonstrated by thelearner 350 in performing the project. In some embodiments, the aspectsor factors of the project may be created by the request modifier 320based on skill maps related to the learner's career interest areas, thatthe learner 350 has received lower scores in the past and/or criteriathe request modifier 320 predicts are relevant based on similar feedbackrequests. The request may request scores and/or comments on each of oneor more aspects or factors of the project.

The request may include a desired volume of feedback. The desired volumeof feedback may indicate a desired number of contributors 360 thatprovide feedback on a project of a learner 350 and/or a quality andquantity of feedback from one or more contributors 360. As anon-limiting example, the desired volume may comprise a request for atleast three contributors to provide feedback comprising rating variousfactors of a project using a supplied rubric. In another embodiment, oneor more types of feedback may have a minimum number of contributorfeedback(s) and/or one or more types of feedback may have a maximumnumber of contributor feedback(s). As an example, the request mayinclude a request for at least two annotated feedbacks and no quickreaction (thumbs up or thumbs down) feedbacks for the project.

The request may also include one or more groups of potentialcontributors 360 that may be contacted and asked to provide feedback tothe learner 350 for the project. The contributors 360 may be organizedinto any desired group or number of groups. As non-limiting examples,the groups available for selection and the selected groups may be fellowlearners, graduate students, other instructors 370, alumni from theonline educational institute 300, alumni from other educationalinstitutes, professional groups (preferably related to the project insome manner), contributor volunteers and/or professionals working in afield related to the project. The request may also include one or morespecific individuals to request for feedback as desired contributors360. In some embodiments, a contributor group request may be required orpreferred. Learners may be given the choice (either always provided oronly after earning a certain status in the system.)

The request may also include a contributor rating, wherein onlycontributors 360 in the selected groups having the contributor rating orhigher receive a request for feedback from the request modifier 320. Inthis manner, the instructor 370 or learner 350 making the request maylimit the feedback to be only from contributors 360 that have previouslybeen rated (possibly by other contributors 360, instructors 370 and/orlearners 350) as having the specified contributor rating. In someembodiments, the request might only include the minimum contributorrating based on a predetermined number of feedback requests of differenttypes.

The contributor rating may be based on any desired metric. Asnon-limiting examples, the contributor rating may be based on metricssuch as: 1) quality of feedback contributions, including learner ratingsand how your feedback relates to the feedback of others with credibilityover time; 2) quantity of feedback contributions over time; 3) types offeedback contributed; 4) feedback types the learners find most usefulare prioritized on an ongoing dynamic basis; 5) a combination ofresponse types would earn status, this may be whatever the system showsstrikes the right balance (for learners and contributors) betweencontributing to the system and with quicker ratings (thumbs up/down) andproviding open-ended response types; 6) demand level of the learner'snetwork, i.e. contributors from more highly requested networks earnhigher status; and 7) willingness to share quote and name of thecontributor 360 to allow the learner 350 to reference that in aportfolio of the learner 350.

The request modifier 320 may assign a character string to the defaultrequest entered by the instructor 370. (Step 600) As an example, theinstructor 370 may enter in the first interface that the feedback duedate is September 1st, the project title is “public relations pitch deckfor new community center,” the suggested feedback type is completing(writing comments in) a suggestions area for the project, the selectedgroups are “graduate students” and “PR professionals” and the desiredvolume of feedback is at least two contributors 360 providing feedback.The request modifier 320 may assign a character string of“PR101Feedback” and display the character string on the client device108 of the instructor 370. The instructor 370 may inform the learners350 in their class, either in person or over their client devices 108,that a default request has been entered into the request modifier 320and the learners 350 can access the default request in the requestmodifier 320 with the character string of “PR101Feedback.” (Step 610)

FIG. 7 illustrates an interface that may be displayed on a client device106 of a learner 350. If the instructor 370 provided a character stringfor a project, the leaner 350 may select, as a non-limiting example, thehyper link 700 so that the learner may enter the character string in afield (such as the pop-up field 800 illustrated in FIG. 8). The learners350, having received the character string for a default request from theinstructor 370, may electronically submit the character string fromtheir client to the request modifier 320. (Step 620) Thus, continuingwith the example, the client device 108 of the learner 350 may transmitthe character string “PR101Feedback” to the request modifier 320.

The request modifier 320 may electronically transmit a second interfaceto the client device 108 of the learner 350. The second interfacepreferably displays to the client device 108 of the learner 350 thedefault request entered by the instructor 370. The default request maybe auto populated into various fields displayed on the second interfaceto the client device 108 of the learner 350. The second interface may beconfigured so that the learner 350 may not change the default request sothat the learner 350 has to use the default request entered by theinstructor 370.

In another embodiment, the second interface may allow the learner 350 tomodify the default request entered by the instructor 370. The secondinterface may allow the client device 108 of the learner 350 to enter aninitial request, different from the default request entered by theinstructor 370, for the feedback for the project from client devices 108of the plurality of contributors 360 to the learner 350. (Step 420) As anon-limiting example, FIG. 7 illustrates an interface that may be usedby a learner 350 to enter an initial request (a similar interface may beused by the instructor 370 to enter the default request). The learner350, using a client device 106, may enter: a project description infield 705; enter or select one or more categories in section or fields710; may enter a course title in field 715; enter or select a projectstatus at menu 720; enter or select a learner confidence level at fieldor menu 725; enter or select a close feedback date in field 730; enteror select one or more career skill focuses of the learner 350 in field740; enter or select one or more preferred or allowed types of feedback,such as quick reaction, rubric-based, descriptive, audio, individual,video, career skills, quick question, annotations on a written projectin section 745; enter a rubric (if one is to be used by contributors 360for the project) in field 750; enter a question to be answered by thecontributors 360 regarding the project in field 755; enter or select oneor more groups from which to select contributors in field 760; enter asearch query to search for a contributor 360 for the project in field765; select zero, one or more recommendations as contributors 360 infield 770; limit the number of selected contributors 360 in filed 775,possibly to those having at least the contributor rating entered orselected in field 780; enter a sharable URL for the contributors 360relevant to the feedback request for the project in field 785; uploadthe project to the request modifier 320 in field 790 to permit access tothe project by the contributors 360; and select submit the initialrequest in field 795 so that the request modifier 320 may submit thecreated and/or modified request to the client devices 106 of thecontributors 360.

In another embodiment, the learner 350 may create an initial requestthat comprises, as non-limiting examples, the feedback due date or theproject due date, the project title or the project description, thesuggested feedback type, the desired volume of feedback and one or moregroups, each group comprising a plurality of potential contributors 360.(Step 400)

In another embodiment the instructor 370 does not create a defaultrequest nor does the instructor 370 provide the learners 350 anassociated character string as each learner 350 from a client device 108may enter and customize an initial request on their own using the secondinterface. In this embodiment, the learner 350 may create an initialrequest that comprises, as non-limiting examples, the feedback due dateor the project due date, the project title or the project description,the suggested feedback type, the desired volume of feedback and one ormore groups, each group comprising a plurality of potential contributors360.

The request modifier 320 may electronically receive a plurality of pastrequests and a corresponding plurality of past feedback from a hardwaredatabase 110. The request modifier 320 may determine a plurality of mostsimilar past requests to the initial request in the plurality of pastrequests. The request modifier 320 may use any desired method ofcomparing and determining which past requests are most similar to theinitial request.

As one possible non-limiting example, the request modifier 320 maydetermine the past requests that are most similar to the initial requestby determining a distance in high dimensional space between the initialrequest and each past request in the plurality of past requests. Forthis example, each factor that may be modified in a request may beassigned a number in a dimension. As non-limiting examples, the due datemay be assigned a number representing the number of days betweensubmitting the request to the contributors 360 and the due date for thefeedback.

The date of the requested feedback may also be assigned a number as theamount of feedback from different contributors 360 may change over time.Thus, the date of each request should also be tracked as a dimension inhigh dimensional space.

The project title or project description may be parsed for subject words(such as marketing) and each subject word may be assigned a number. Inpreferred embodiments, subjects that are closer to each other areassigned numbers that are closer to each other.

The suggested feedback may be assigned the number one for a scoringmethod of a positive or a negative reaction (thumbs up or thumbs down),a two for a rating on a scale, a three for ratings to various factorslisted in a rubric, a four for a drafting comments in a suggestions areaand a five for an annotated response to the material of the project. Insome embodiments, the comments may be in a video and/or an audio format.If audio is used, the audio may be converted to text which could bedisplayed to the learner 350 as comments. Alternatively, or in addition,the video may also be converted to text which could be displayed to thelearner 350.

The one or more selected contributor groups may be assigned a one forfellow learners, a two for graduate students, a three for otherinstructors 370, a four for alumni from the online educational institute300, a five for alumni from other educational institutes, a six forprofessional groups (preferably related to the project in some manner),and a seven for professionals working in a field related to the project.If multiple groups are selected, as a non-limiting example, their valuesmay be added together or a plurality (such as four) of dimensions may bereserved for the groups selected. If fewer groups are selected than thenumber of dimensions reserved for groups, not used dimensions may beassigned a default value, such as zero.

The order of each factor as a dimension may be any desired order, butthe order should be kept consistent between the current requests and allpast requests. In this manner, the current initial request and all pastrequests may be placed in high dimensional space.

The request modifier 320 may select as the plurality of most similarpast requests the past requests in the plurality of past requests (thathave been saved in the hardware database 110) with the smallestdetermined distance in high dimensional space with the initial request.The distance between the initial request and past requests may becalculated by subtracting their positions from each other in highdimensional space.

While any number of similar past requests may be selected, as a specificexample, the 10 closest past requests to the initial request may beselected and their volumes of feedback (number of contributors 360,quality ratings and/or quantity of feedback) may be averaged. Theaverage volume of feedback for the closest past requests to the initialrequest may be taken as a predicted volume of feedback for the initialrequest.

The request modifier 320 may assign a predicted volume of feedback forthe initial request based on an average volume of past feedback for theplurality of most similar past requests. This method assumes thatsimilar requests will receive similar volumes of feedback.

In some embodiments, the request modifier 320 may modify the initialrequest to generate a modified request so that the modified request ispredicted to receive a volume of feedback closer to the desired volumeof feedback than the predicted volume of feedback for the initialrequest.

As an example, if the desired volume of feedback is greater than thepredicted volume of feedback for the initial request, the initialrequest may be modified by any desired means that is likely to increasethe volume of predicted feedback. The initial request may be modified toreceive more feedback by, as non-limiting examples, selecting a largergroup, increasing the number of groups selected and/or selecting groupsthat are known to have more contributors 360 that are likely tocontribute feedback. As another non-limiting example, the initialrequest may be modified so that the requested type of feedback is easierto complete. Thus, an initial request asking for annotated comments maybe modified to ask for a quick reaction, such as a thumbs up or a thumbsdown assuming more contributors 360 are likely to participate if therequired feedback is easier or quicker to produce by the contributors360. In some embodiments, the modified request may be analyzed as theinitial request was analyzed to determine a predicted volume of feedbackfor the modified request. The modified request in this embodiment may berepeatedly modified until the predicted volume of feedback for themodified request is within a predetermined distance from the desiredvolume of feedback or a predetermined time or number of modificationshave been attempted.

As another example, if the desired volume of feedback is less than thepredicted volume of feedback for the initial request, the initialrequest may be modified by any desired means that is likely to decreasethe volume of predicted feedback. The initial request may be modified toreceive less feedback by, as non-limiting examples, selecting a smallergroup, decreasing the number of groups selected and/or selecting groupsthat are known to have fewer contributors 360 that are likely tocontribute feedback. As another non-limiting example, the initialrequest may be modified so that the requested type of feedback is harder(but better for the educational experience of the learner 350) tocomplete. Thus, an initial request asking for a thumbs up or thumbs downreaction may be changed to request annotated comments. In someembodiments, the modified request may be analyzed as the initial requestwas analyzed to determine a predicted volume of feedback for themodified request. The modified request in this embodiment may berepeatedly modified until the predicted volume of feedback for themodified request is within a predetermined distance from the desiredvolume of feedback or a predetermined time or number of modificationshave been attempted. It should be appreciated that the request modifier320 may be automatically making the modifications to the modifiedrequest with or without approval from the learner 350, possibly based onsettings selected or approved by the learner 350.

In preferred embodiments, if the default request or the initial requestfor feedback entered by the learner 350 is modified, the modifiedrequest, preferably highlighting the changes, may be displayed to thelearner 350 for the learner's approval. In preferred embodiments, a highlevel explanation for the reason for the change may be displayed to thelearner 350. If the learner 350 does not approve the modified request,the original default request or the initial request may be used forsubmittal to the selected group(s) of contributors 360. If the learner350 does approve the modified request, the modified request may be usedfor submittal to the selected group(s) of contributors 360. (Step 420)

It should be noted that in some embodiments the instructor's defaultrequest must be used by the learners 350. In other embodiments, theinstructor's default request may be modified by the learners 350. Inother embodiments, the instructor 370 does not enter a default requestand the learner 350 may select the initial request for feedback from thecontributors 360 using menus, fields and/or selection tabs on aninterface displayed on the client of the learner 350. In preferredembodiments, the instructor 370 is able to select whether the defaultrequest entered by the instructor 370 is mandatory, the default requestmay be modified or the learners 350 must enter their own initialrequest.

It should be appreciated that in some embodiments the default requestentered by the instructor 370 may be used without being modified (oreven attempted to be modified) to receive a desired volume of feedbackand/or modified by the learners 350. In other embodiments, an initialrequest entered by a learner 350 may be used without being modified (oreven attempted to be modified) to receive a desired volume of feedbackfrom the contributors 360. Thus, the request modifier 320 mayelectronically transmit the default request, the initial request or themodified request to the selected contributors 360, thereby requestingfeedback from the contributors 360 for the project of the learner 350.(Step 430)

In a preferred embodiment, each selected contributor 106 is given anoption to accept or reject a request for feedback from the learner 350regarding a project. As an example, FIG. 9 illustrates an interfacecommunicated to a client device 106 of a contributor 360 that displaysthe name (and optionally picture or video) of the learner 910, a projecttitle or stage of the project 920, a due date for the feedback on theproject 930, and feedback request instructions regarding the type offeedback the learner 350 is requesting from the contributor 360 as shownin field 940. The interface may also have an accept button 950 and areject button 960 so that the contributor 360 may easily accept orreject the request to provide feedback for the project. In this manner,the request modifier 320 may send requests to additional contributors370 if too many of the contributors 370 are rejecting the request forfeedback for the project for the learner 350.

Each contributor 360 preferably has options, via settings thecontributor 360 has selected, to control the types, frequency, volumeand request categories and characteristics (such as institution, year inschool, major(s), subject areas, etc.) that the contributor 360 iswilling to provide feedback. The system preferably does not requestfeedback from a contributor 360 that is not within the settings orbounds selected by the contributor 360. In other embodiments, eachcontributor 360 may search for outstanding feedback requests andvolunteer to provide feedback for as few or as many feedback requests asdesired by the contributor 360 (as long as the contributor 360 meets theminimum qualifications as requested by the learner 350).

In preferred embodiments, the selected contributors 360 that havereceived a request for feedback may receive an interface from which toreview the project of the learner 350 and from which to enter feedbackregarding the project of the learner 350. The interface preferablyinforms the contributors 360 of the date by which the feedback must beentered and the type of feedback desired by the instructor 370 orlearner 350, e.g., a scoring method of a positive or a negative reaction(thumbs up or thumbs down), a rating on a scale, a ratings for variousfactors listed in a displayed rubric in the interface, comments in asuggestions area or an annotated response to the material of theproject. In some embodiments, only means for entering the desired typeof feedback are presented to the contributors 360 on the interface.

FIG. 10 illustrates an interface for a contributor 360 to enterannotated comments in field 106. The contributor 360 may be identifiedin field 101, a project rating may be identified in field 1020, commentsmay be entered by the contributor 360 in field 1030, the skills asselected by the contributor 360 may be displayed in field 1040 and oneor more predefined action items may be displayed in field 1050 for thecontributor 360 to select, thereby making the review and feedbackprocess for the contributor 360 easier to complete.

If the type of requested feedback is rating various factors listed in arubric, the rubric may be displayed on the interface with means ofrating (such as selecting a number or letter on a scale) each factor(such as creativity or writing skill) listed in the rubric, but in thisembodiment other rating methods would not be displayed on the interface.In this manner a feedback aggregator 330 may electronically receive thedesired type of feedback from the clients of the plurality ofcontributors 360. (Step 440)

FIG. 11 illustrates an interface that may be displayed on a clientdevice 106 of a learner 350 that displays the feedback from one or morecontributors. In this example interface 1100, three contributors 360 arelisted (John Li, Matt Smith and Ella Bakubi), each with differentcontributor ratings and using different types of feedback. The interfacealso provides statistics on the average rating from the contributors 360and the response rate of the contributors 360 for the project of thelearner 350.

The feedback aggregator 330 may perform various statistical analysis onthe feedback, such as percentage of contributors 360 receiving thesubmitted request that actually provided feedback and/or the averagescores of the contributors 360 for the project. The feedback aggregator330 may also combine all the comments or suggestions from the differentcontributors 360 onto a single interface or web page, for easy viewingon a client device 106 by the instructor 370 and/or learner 350. Thefeedback aggregator 330 may make available and display the feedback tothe instructor 370 and learner 350 by electronically transmitting aninterface displaying the plurality of feedback from the plurality ofcontributors 360 to the client device 106 of the learner 350 and aclient device 106 of an instructor 370 for the project. (Step 450) Insome embodiments, the learner 350 may tag one or more feedbacks from thecontributors (or portions thereof) which may be automatically sharedwith the instructor 370.

FIG. 12 illustrates an interface 1200 that lists when the feedback isneeded in field 1210, the selected one or more groups listed in field1220, a description of the project in field 1230, the suggested feedbacktype(s) in field 1240, a project description in field 1250, a feedbackfocus listed in field 1260, a career skills listed in field 1270, and aplurality of feedback from contributors 360 in field 1280. Graphs 1290may also be displayed showing the ratings from contributors 360 forvarious skills of the learner 350 displayed in the performance of theproject. Graphs 1290 may also represent scores for all projectssubmitted by the learner 350 with feedback ratings tied to that skill.In addition or alternatively, a graph may also be used to visualize thedata for each project's related career.

In preferred embodiments, each learner 350 only receives an interfacethat has feedback specifically for the project of the learner 350. Inother words, learners 350 preferably do not see feedback intended forother learners 350. In some embodiments, for a common project performedby a group, i.e., a plurality of learners 350, each member of the groupmay see feedback regarding the common project. The instructor 370 ispreferably able to see the feedback received by all of the learners 350for the project in the instructor's class. In other embodiments, thelearner 350 and/or contributors 370 may have to affirmatively givepermission to the instructor 370 before the instructor may see thecontributors' feedback.

FIG. 13 illustrates an interface 1300 displaying feedback summary for aplurality of past projects 1330 and 1340. In this example illustrationelectronically communicated to the client device 106 of the learner 350,a feedback status 1310 is displayed showing, as non-limiting examples, apercentage of overall feedback rate, a percentage of responses frompreferred networks, a number of feedback requests, a number of projectviews and a number of total and starred contributors 360.

The feedback aggregator 330 may electronically store the submittedrequest (the request actually electronically sent to the contributors360) and the plurality of feedback from the plurality of contributors360 or a summary of the feedback in the hardware database 110. Thus,over time the request modifier 320 and feedback aggregator 330 of theonline educational institute 300 will have access to an ever increasinghardware database 110 of submitted requests and the feedback receivedfrom those submitted requests which may be used to modify future initialrequests to get improved feedback, such as the type of desired feedbackor the volume of desired feedback for the learners 350.

In another embodiment, the feedback from each contributor for eachlearner 350 may be used to calculate various skill levels of the learner350. As non-limiting examples, the skills of writing and/or creativitymay be evaluated and scored by a plurality of contributors 360. Thescores may be normalized based on the past scores of the contributors360 (thus learners 350 with harder grading contributors 360 are notunfairly punished and learners 350 with easier grading contributors 360are not unfairly rewarded) and averaged to give each learner 350 a scorefor a plurality of different skills. The scores and associated skillsmay be displayed to the learner 350.

As a non-limiting example, FIG. 14 illustrates an interface 1400 thatdisplays the scores for various skills of the learner 350 with thevarious skills necessary for a particular career or job title. Aplurality of normalized scores for the learner 350 may be comparedagainst scores necessary, suggested or recommended for differentemployment positions, careers or job titles. Thus, if the learner 350wants a particular career of marketing director, but has a creativewriting score below a listed necessary, suggested or recommendedcreative writing score for the position of a marketing director, thelearner 350 may focus on that particular skill in the future. In otherembodiments, the system may also suggest that a learner include a skill(in the feedback request criteria) that the learner is low in or thatthe system does not have enough data on yet. In addition, the learner350 may discover other employment positions, careers or job titles thatmatch closely or where the learner 350 exceeds the necessary, suggestedor recommended scores and thus may learn of a new career path that maybe well suited to the learner 350 that the learner 350 had notconsidered before and then track skills and pursue feedback on theseattributes over time.

The career skills component of the invention may be used to helplearners 350 track their progress towards their targeted careers. Thecareer skills component may also match learners 350, based on feedbackdata, and usage analytics, to careers that the learners 350 may not haveconsidered or targeted previously. The career skills component mayreference visualizations of skills needed in a given career and trackand promote learner 350 progress towards those skills. Learner 350skills relative to desired level on the career skills graph will beearned through feedback requests that include rating specific skills thelearner needs and responses that address specific skills (for examplevia NLP). Just as in gaming, a learner 350 may start with zero points onthe career skills graph and earn progress on it based on feedbackrequests and metrics. Skills (perhaps the top 2 or 3) for which moreevidence is needed will be suggested as a default when the learner 350submits a new request; there may be an option to modify the new requestbefore sending the request out. Progress along a certain dimension ofthe career skills graph may be calculated based on metrics such asratings and responses specific to a given skill (e.g., critical thinkingas a criterion within a rubric) and the credibility and status of thoseproviding the feedback.

Other embodiments and uses of the above inventions will be apparent tothose having ordinary skill in the art upon consideration of thespecification and practice of the invention disclosed herein. Thespecification and examples given should be considered exemplary only,and it is contemplated that the appended claims will cover any othersuch embodiments or modifications as fall within the true scope of theinvention.

The Abstract accompanying this specification is provided to enable theUnited States Patent and Trademark Office and the public generally todetermine quickly from a cursory inspection the nature and gist of thetechnical disclosure and in no way intended for defining, determining,or limiting the present invention or any of its embodiments.

The invention claimed is:
 1. A method for an online digital feedbacknetwork to provide a plurality of online digital feedback for a projectfrom a plurality of contributors to a feedback recipient, comprising thesteps of: electronically receiving over the Internet by a requestmodifier, comprising computer code running on one or more hardwareservers operated by the online digital feedback network, an initialrequest from a client device of a feedback recipient for online digitalfeedback from client devices of the plurality of contributors;electronically receiving by the request modifier from the client deviceof the feedback recipient a desired volume of online digital feedbackfrom the client devices of the plurality of contributors; electronicallyreceiving by the request modifier a plurality of past requests and acorresponding plurality of past online digital feedback from a hardwaredatabase; determining by the request modifier a plurality of mostsimilar past requests to the initial request in the plurality of pastrequests; assigning by the request modifier a predicted volume of onlinedigital feedback for the initial request based on an average volume ofpast online digital feedback for the plurality of most similar pastrequests; automatically modifying by the request modifier the initialrequest to generate a modified request, wherein the modified request ispredicted to receive a volume of online digital feedback closer to thedesired volume of online digital feedback than the predicted volume ofonline digital feedback for the initial request; electronicallytransmitting by the request modifier the modified request to theplurality of contributors for online digital feedback of the project ofthe feedback recipient, wherein each contributor receives a userinterface from which to accept the modified request, review the projectof the feedback recipient and from which to enter online digitalfeedback regarding the project of the feedback recipient; electronicallyreceiving by a feedback aggregator, comprising computer code running onthe one or more hardware servers, the plurality of online digitalfeedback from the plurality of contributors; and electronicallytransmitting by the feedback aggregator a user interface displaying theplurality of online digital feedback from the plurality of contributorsto the client device of the feedback recipient and a client device of adata source for the project.
 2. The method of claim 1, furthercomprising the step of: electronically storing by the feedbackaggregator the submitted request and the plurality of online digitalfeedback from the plurality of contributors or a summary of the onlinedigital feedback in the hardware database.
 3. The method of claim 1,wherein the step of determining by the request modifier the plurality ofmost similar past requests to the initial request in the plurality ofpast requests comprises the steps of: determining a distance in highdimensional space between the initial request and each past request inthe plurality of past requests; and selecting as the plurality of mostsimilar past requests the past requests in the plurality of pastrequests with the smallest determined distance in high dimensional spacewith the initial request.
 4. The method of claim 1, further comprisingthe step of: electronically receiving by the request modifier anapproval of the modified request from the client device of the feedbackrecipient.
 5. The method of claim 1, wherein the initial request fromthe client device of the feedback recipient comprises a feedback duedate or a project due date, a project title or a project description, asuggested feedback type and a plurality of groups, each group comprisinga plurality of potential contributors.
 6. The method of claim 5, whereinthe initial request further comprises a contributor rating, wherein onlycontributors in the plurality of groups having the contributor rating orhigher receive the modified request for online digital feedback from therequest modifier.
 7. The method of claim 5, wherein the suggestedfeedback type comprises a positive or negative reaction or an annotatedresponse.
 8. The method of claim 5, wherein the step of modifying by therequest modifier the initial request to generate a modified requestcomprises adding a group to the modified request to increase the volumeof online digital feedback or removing a group from the modified requestto decrease the volume of online digital feedback.
 9. A method for anonline digital feedback network to provide a plurality of online digitalfeedback for a project from a plurality of contributors to a feedbackrecipient, comprising the steps of: electronically transmitting by arequest modifier, comprising computer code running on one or morehardware servers, a first user interface to a client device of a datasource configured to allow the client device of the data source to entera default request for the online digital feedback for the project fromclient devices of the plurality of contributors to the feedbackrecipient, wherein the default request comprises a feedback due date ora project due date, a project title or a project description, asuggested feedback type and a plurality of groups, each group comprisinga plurality of potential contributors; assigning by the request modifiera character string to the default request entered by the data source;electronically receiving by the request modifier the character string tothe default request from a client device of the feedback recipient;electronically transmitting by the request modifier a second userinterface to the client device of the feedback recipient configured toallow the client device of the feedback recipient to enter an initialrequest for the online digital feedback for the project from clientdevices of the plurality of contributors to the feedback recipient,wherein the initial request comprises the feedback due date or theproject due date, the project title or the project description, thesuggested feedback type and the plurality of groups, each groupcomprising a plurality of potential contributors; electronicallytransmitting by the request modifier the initial request to theplurality of contributors for online digital feedback, wherein eachcontributor receives an interface from which to accept the initialrequest, review the project of the feedback recipient and from which toenter online digital feedback regarding the project of the feedbackrecipient; electronically receiving by a feedback aggregator, comprisingcomputer code running on one or more hardware servers, the plurality ofonline digital feedback from the plurality of contributors for theproject of the feedback recipient; and electronically transmitting bythe feedback aggregator a third user interface displaying the pluralityof online digital feedback from the plurality of contributors to theclient device of the feedback recipient and the client device of thedata source for the project.
 10. The method of claim 9, furthercomprising the step of: electronically storing by the feedbackaggregator the submitted request and the plurality of online digitalfeedback from the plurality of contributors or a summary of the onlinedigital feedback in the hardware database.
 11. The method of claim 9,wherein the step of determining by the request modifier a plurality ofmost similar past requests to the initial request in the plurality ofpast requests comprises the steps of: determining a distance in highdimensional space between the initial request and each past request inthe plurality of past requests; and selecting as the plurality of mostsimilar past requests the past requests in the plurality of pastrequests with the smallest determined distance in high dimensional spacewith the initial request.
 12. The method of claim 9, further comprisingthe step of: electronically receiving by the request modifier anapproval of the initial request from the client device of the feedbackrecipient.
 13. The method of claim 9, wherein the initial requestfurther comprises a contributor rating, wherein only contributors in theplurality of groups having the contributor rating or higher receive theinitial request for online digital feedback from the request modifier.14. The method of claim 9, further comprising the step of modifying bythe request modifier the initial request to generate a modified requestcomprises adding a group to the modified request to increase the volumeof online digital feedback or removing a group from the modified requestto.
 15. A method for an online digital feedback network to provide aplurality of online digital feedback for a project from a plurality ofcontributors to a feedback recipient, comprising the steps of:electronically receiving by a request modifier, comprising computer coderunning on one or more hardware servers operated by the online digitalfeedback network, an initial request from a client device of a feedbackrecipient for online digital feedback from client devices of theplurality of contributors; electronically receiving by the requestmodifier from the client device of the feedback recipient a desiredvolume of online digital feedback from the client devices of theplurality of contributors; electronically receiving by the requestmodifier a plurality of past requests and a corresponding plurality ofpast online digital feedback from a hardware database; determining bythe request modifier a plurality of most similar past requests to theinitial request in the plurality of past requests; assigning by therequest modifier a predicted volume of online digital feedback for theinitial request based on an average volume of past online digitalfeedback for the plurality of most similar past requests; automaticallymodifying by the request modifier the initial request to generate amodified request, wherein the modified request is predicted to receive avolume of online digital feedback closer to the desired volume of onlinedigital feedback than the predicted volume of online digital feedbackfor the initial request; electronically transmitting by the requestmodifier the modified request to the plurality of contributors foronline digital feedback of the project of the feedback recipient,wherein each contributor receives a user interface from which to acceptthe modified request, review the project of the feedback recipient andfrom which to enter online digital feedback regarding the project of thefeedback recipient; electronically receiving by a feedback aggregator,comprising computer code running on the one or more hardware servers,the plurality of online digital feedback from the plurality ofcontributors; electronically transmitting by the feedback aggregator auser interface displaying the plurality of online digital feedback fromthe plurality of contributors to the client device of the feedbackrecipient; and electronically storing by the feedback aggregator thesubmitted request and the plurality of online digital feedback from theplurality of contributors or a summary of the online digital feedback inthe hardware database.
 16. The method of claim 15, wherein the step ofdetermining by the request modifier the plurality of most similar pastrequests to the initial request in the plurality of past requestscomprises the steps of: determining a distance in high dimensional spacebetween the initial request and each past request in the plurality ofpast requests; and selecting as the plurality of most similar pastrequests the past requests in the plurality of past requests with thesmallest determined distance in high dimensional space with the initialrequest.
 17. The method of claim 15, further comprising the step of:electronically receiving by the request modifier an approval of themodified request from the client device of the feedback recipient. 18.The method of claim 15, wherein the initial request from the clientdevice of the feedback recipient comprises a feedback due date or aproject due date, a project title or a project description, a suggestedfeedback type and a plurality of groups, each group comprising aplurality of potential contributors.
 19. The method of claim 15, whereinthe initial request further comprises a contributor rating, wherein onlycontributors in the plurality of groups having the contributor rating orhigher receive the modified request for online digital feedback from therequest modifier.